Self-Driving Car Engineer Nanodegree¶
Deep Learning¶
Project: Build a Traffic Sign Recognition Classifier¶
In this notebook, a template is provided for you to implement your functionality in stages, which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission if necessary.
Note: Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.
In addition to implementing code, there is a writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a write up template that can be used to guide the writing process. Completing the code template and writeup template will cover all of the rubric points for this project.
The rubric contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this Ipython notebook and also discuss the results in the writeup file.
Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.
Step 0: Load The Data¶
In [1]:
enable_aug = False
# Load pickled data
import pickle
# TODO: Fill this in based on where you saved the training and testing data
training_file = "./traffic-signs-data/train.p"
validation_file="./traffic-signs-data/valid.p"
testing_file = "./traffic-signs-data/test.p"
with open(training_file, mode='rb') as f:
train = pickle.load(f)
with open(validation_file, mode='rb') as f:
valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
test = pickle.load(f)
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']
Step 1: Dataset Summary & Exploration¶
The pickled data is a dictionary with 4 key/value pairs:
'features'is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).'labels'is a 1D array containing the label/class id of the traffic sign. The filesignnames.csvcontains id -> name mappings for each id.'sizes'is a list containing tuples, (width, height) representing the original width and height the image.'coords'is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES
Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the pandas shape method might be useful for calculating some of the summary results.
Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas¶
In [2]:
### Replace each question mark with the appropriate value.
### Use python, pandas or numpy methods rather than hard coding the results
import numpy as np
# TODO: Number of training examples
n_train = X_train.shape[0]
# TODO: Number of testing examples.
n_valid = X_valid.shape[0]
# TODO: Number of testing examples.
n_test = X_test.shape[0]
# TODO: What's the shape of an traffic sign image?
image_shape = X_train.shape[1:3]
# TODO: How many unique classes/labels there are in the dataset.
n_classes = np.unique(y_train)
print("Number of training examples =", n_train)
print("Number of valid examples =", n_valid)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Include an exploratory visualization of the dataset¶
Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.
The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.
NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.
In [3]:
### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline
## import name of each class
import csv
classes_string = []
with open('signnames.csv', newline='') as csvfile:
rd = csv.reader(csvfile, delimiter=',', quotechar='|')
i = 0
for row in rd:
if i==0:
i += 1
continue
else:
classes_string.append(row[1])
fig, ax = plt.subplots(figsize=(16,4))
n_bins = 43
ax.hist(y_train,n_bins, histtype='bar', color='green', label='train')
ax.hist(y_test,n_bins, histtype='bar', color='blue', label='test')
ax.hist(y_valid,n_bins, histtype='bar', color='magenta', label='valid')
ax.set_xticks(range(n_bins))
ax.set_xticklabels(classes_string, rotation=40,ha='right')
ax.set_title('Data ditribution (train, test, valid)')
plt.legend(['train','test','valid'])
Out[3]:
Sample visualization of each class¶
In [7]:
# show data distributions of train, valid, test samples
import numpy as np
import matplotlib.pyplot as plt
def GetSampleData(data, labels, n_class):
import random as rm
n_data = data[ labels == n_class ]
if n_data.shape[0]==0:
return np.array([])
else:
sample = n_data[ rm.randint(0, n_data.shape[0]-1) ]
return sample.squeeze()
def GetSampleGallary(data, labels, n_class, num):
if num < 1:
print('please insert above 1 for num')
return np.array([])
gallary = np.array([])
for i in range(num):
sample = GetSampleData(data, labels, n_class)
if sample.shape[0] != 0:
if i==0:
gallary = sample
else:
gallary = np.hstack((gallary,sample))
return gallary
fig = plt.figure(figsize=(15,80))
for i in np.unique(y_train):
samples = GetSampleGallary(X_train, y_train, i, 10)
ax = fig.add_subplot(43, 1, i+1)
ax.axis('off')
ax.imshow(samples)
ax.set_title(classes_string[i],fontsize=12)
plt.show()
Pre-process the Data Set¶
Five steps for data augmentation and pre-process to improve classification performance¶
1. Flipping¶
- Horizontal flipping: class number 11, 12, 13, 15, 17, 18, 22, 26, 30, 35
- Vertical flipping: class number 1, 5, 12, 15, 17
- Horizontal and vertcal flipping: 32, 40
- Horizontal flipping, but changed class number
- from: 19,33,36,38,20,34,37,39
- to : 20,34,37,39,19,33,36,38
2. Rotation¶
- Random rotation within +-15 degree was applied to the flipped samples
3. Perspective transform¶
- Perspective transform was conducted with 5 iterations
- Four corners were randomly selected in certain boundaries
4. Y tranformation¶
- RGB image was converted to YUV color space
- Only Y channel was saved for training samples
5. Equalization¶
- Histogram equalization was applied to the Y channel
In [229]:
### Function definition for data augmentation
## normalization
def normalized_gray(x, alpha, beta):
x_mean = np.ceil((x[:,:,:,0] + x[:,:,:,1] + x[:,:,:,2])/3)
x = alpha + (x_mean - 0) / (255 - 0 ) * (beta - alpha)
x.shape = (x.shape[0], 32, 32, 1)
return x
def intensity_normalization(x):
import cv2
dst = []
for img in x:
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img_gray = img_gray / (np.max(img_gray))
img_gray.shape = (32,32,1)
dst.append(img_gray)
dst = np.array(dst)
return dst
def convertRGBtoY(x):
x = 0.299 * x[:,:,:,0] + 0.587 * x[:,:,:,1] + 0.114 * x[:,:,:,2]
x = (x / 255.).astype(np.float32)
return x
def normalize(x):
return (x/255.0).astype('float32')
def equalize_hist(x):
from skimage import exposure
import os
for i in range(len(x)):
x[i] = exposure.equalize_adapthist(x[i])
if (i/len(x)*100) % 10 == 0:
print('Progress: {:.2f}%'.format(i/len(x)*100))
return x
def flip_h(x):
return x[:,:,::-1,:]
def flip_v(x):
return x[:,::-1,:,:]
def flip_hv(x):
return x[:,::-1,::-1,:]
## translation, scaling, rotation, flipping
def flip_data(x, y, labels, cross_labels=None, method=0):
_x = np.empty([0,x.shape[1],x.shape[2],x.shape[3]], dtype=x.dtype)
_y = np.empty([0], dtype=y.dtype)
for i in range(len(labels)):
x_label = x[y==labels[i]]
y_label = y[y==labels[i]]
if method==0: # flip horizontally
x_label_flipped = flip_h(x_label)
y_label_flipped = (np.full((len(x_label_flipped)),labels[i])).astype(y.dtype)
elif method==1: # flip vertically
x_label_flipped = flip_v(x_label)
y_label_flipped = (np.full((len(x_label_flipped)),labels[i])).astype(y.dtype)
elif method==2: # flip both
x_label_flipped = flip_hv(x_label)
y_label_flipped = (np.full((len(x_label_flipped)),labels[i])).astype(y.dtype)
elif method==3: # flip vertically and cross label
x_label_flipped = flip_h(x_label)
y_label_flipped = (np.full((len(x_label_flipped)),cross_labels[i])).astype(y.dtype)
_x = np.append(_x,x_label_flipped, axis=0)
_y = np.append(_y,y_label_flipped, axis=0)
return _x, _y
def flip_data_set(x, y, f_list):
x_flipped = np.empty([0,x.shape[1],x.shape[2],x.shape[3]],dtype=x.dtype)
y_flipped = np.empty([0],dtype=x.dtype)
_x, _y = flip_data(x, y, f_list["h"], None, method=0)
x_flipped=np.append(x_flipped,_x,axis=0)
y_flipped=np.append(y_flipped,_y,axis=0)
_x, _y = flip_data(x, y, f_list["v"], None, method=1)
x_flipped=np.append(x_flipped,_x,axis=0)
y_flipped=np.append(y_flipped,_y,axis=0)
_x, _y = flip_data(x, y, f_list["hv"], None, method=2)
x_flipped=np.append(x_flipped,_x,axis=0)
y_flipped=np.append(y_flipped,_y,axis=0)
_x, _y = flip_data(x, y, f_list["cs_h"][0,:], f_list["cs_h"][1,:], method=3)
x_flipped=np.append(x_flipped,_x,axis=0)
y_flipped=np.append(y_flipped,_y,axis=0)
return x_flipped, y_flipped
def rotate_data(X, ang_margin=15.):
from skimage.transform import rotate
from skimage.transform import warp
from skimage.transform import ProjectiveTransform
print(X.shape)
_X = np.empty([0,X.shape[1],X.shape[2],X.shape[3]],dtype=X.dtype)
for i in range(X.shape[0]):
img = rotate(X[i], np.random.uniform(-ang_margin, ang_margin), mode = 'edge')
img = (img * 255).astype('uint8')
img.shape = (1,img.shape[0],img.shape[1],img.shape[2])
_X = np.append(_X,img, axis=0)
return _X
def pespective_transform(X, offset):
image_size = X.shape[1]
_X = np.empty([0,X.shape[1],X.shape[2],X.shape[3]],dtype=X.dtype)
for i in range(X.shape[0]):
tl_top = np.random.uniform(-offset, offset) # Top left corner, top margin
tl_left = np.random.uniform(-offset, offset) # Top left corner, left margin
bl_bottom = np.random.uniform(-offset, offset) # Bottom left corner, bottom margin
bl_left = np.random.uniform(-offset, offset) # Bottom left corner, left margin
tr_top = np.random.uniform(-offset, offset) # Top right corner, top margin
tr_right = np.random.uniform(-offset, offset) # Top right corner, right margin
br_bottom = np.random.uniform(-offset, offset) # Bottom right corner, bottom margin
br_right = np.random.uniform(-offset, offset) # Bottom right corner, right margin
transform = ProjectiveTransform()
transform.estimate(np.array((
(tl_left, tl_top),
(bl_left, image_size - bl_bottom),
(image_size - br_right, image_size - br_bottom),
(image_size - tr_right, tr_top)
)), np.array((
(0, 0),
(0, image_size),
(image_size, image_size),
(image_size, 0)
)))
img = warp(X[i], transform, output_shape=(image_size, image_size), order = 1, mode = 'edge')
img = (img * 255).astype('uint8')
img.shape = (1,img.shape[0],img.shape[1],img.shape[2])
_X = np.append(_X,img, axis=0)
return _X
In [171]:
f_list = {"h": np.array([11, 12, 13, 15, 17, 18, 22, 26, 30, 35]),
"v": np.array([1, 5, 12, 15, 17]),
"hv": np.array([32, 40]),
"cs_h": np.array([[19,33,36,38,20,34,37,39],[20,34,37,39,19,33,36,38]])}
- Data Augementation for Validation Set¶
In [238]:
X_tt = X_valid
y_tt = y_valid
X_flipped, y_flipped = flip_data_set(X_tt,y_tt,f_list)
X_ext = np.append(X_tt,X_flipped,axis=0)
y_ext = np.append(y_tt,y_flipped,axis=0)
print('[flipped]: original', X_tt.shape, '/ extend', X_ext.shape)
X_rotated = rotate_data(X_ext, 15.)
X_ext = np.append(X_ext,X_rotated,axis=0)
y_ext = np.append(y_ext,y_ext,axis=0)
print('[rotated]: ', X_ext.shape)
_x = np.copy(X_ext)
_y = np.copy(y_ext)
for i in range(5):
X_aug = pespective_transform(_x, 5)
X_ext = np.append(X_ext,X_aug,axis=0)
y_ext = np.append(y_ext,_y,axis=0)
print('[pespective transform]: ', X_ext.shape)
X_valid_ext = convertRGBtoY(X_ext)
X_valid_ext = equalize_hist(X_valid_ext)
y_valid_ext = y_ext
print('finished!!')
fig = plt.figure(figsize=(15,80))
for i in np.unique(y_ext):
samples = GetSampleGallary(X_valid_ext, y_ext, i, 10)
ax = fig.add_subplot(43, 1, i+1)
ax.axis('off')
ax.imshow(samples,cmap='gray')
ax.set_title(classes_string[i],fontsize=12)
plt.show()
print(X_valid_ext.shape, y_valid_ext.shape)
np.save('X_valid_ext',X_valid_ext)
np.save('y_valid_ext',y_valid_ext)
- Data Augmentation for Training Set and Sample Visualization¶
In [243]:
X_tt = X_train
y_tt = y_train
X_flipped, y_flipped = flip_data_set(X_tt,y_tt,f_list)
X_ext = np.append(X_tt,X_flipped,axis=0)
y_ext = np.append(y_tt,y_flipped,axis=0)
print('[flipped]: original', X_tt.shape, '/ extend', X_ext.shape)
X_rotated = rotate_data(X_ext, 15.)
X_ext = np.append(X_ext,X_rotated,axis=0)
y_ext = np.append(y_ext,y_ext,axis=0)
print('[rotated]: ', X_ext.shape)
_x = np.copy(X_ext)
_y = np.copy(y_ext)
for i in range(5):
X_aug = pespective_transform(_x, 5)
X_ext = np.append(X_ext,X_aug,axis=0)
y_ext = np.append(y_ext,_y,axis=0)
print('[pespective transform]: ', X_ext.shape)
X_train_ext = convertRGBtoY(X_ext)
X_train_ext = equalize_hist(X_train_ext)
y_train_ext = y_ext
print('finished!!')
fig = plt.figure(figsize=(15,80))
for i in np.unique(y_ext):
samples = GetSampleGallary(X_train_ext, y_train_ext, i, 10)
ax = fig.add_subplot(43, 1, i+1)
ax.axis('off')
ax.imshow(samples,cmap='gray')
ax.set_title(classes_string[i],fontsize=12)
plt.show()
print(X_train_ext.shape, y_train_ext.shape)
np.save('X_train_ext',X_train_ext)
np.save('y_train_ext',y_train_ext)
- Data Augmentation for Test Set and Sample Visualization¶
In [244]:
X_tt = X_test
y_tt = y_test
X_flipped, y_flipped = flip_data_set(X_tt,y_tt,f_list)
X_ext = np.append(X_tt,X_flipped,axis=0)
y_ext = np.append(y_tt,y_flipped,axis=0)
print('[flipped]: original', X_tt.shape, '/ extend', X_ext.shape)
X_rotated = rotate_data(X_ext, 15.)
X_ext = np.append(X_ext,X_rotated,axis=0)
y_ext = np.append(y_ext,y_ext,axis=0)
print('[rotated]: ', X_ext.shape)
_x = np.copy(X_ext)
_y = np.copy(y_ext)
for i in range(5):
X_aug = pespective_transform(_x, 5)
X_ext = np.append(X_ext,X_aug,axis=0)
y_ext = np.append(y_ext,_y,axis=0)
print('[pespective transform]: ', X_ext.shape)
X_test_ext = convertRGBtoY(X_ext)
X_test_ext = equalize_hist(X_test_ext)
y_test_ext = y_ext
print('finished!!')
fig = plt.figure(figsize=(15,80))
for i in np.unique(y_ext):
samples = GetSampleGallary(X_test_ext, y_test_ext, i, 10)
ax = fig.add_subplot(43, 1, i+1)
ax.axis('off')
ax.imshow(samples,cmap='gray')
ax.set_title(classes_string[i],fontsize=12)
plt.show()
print(X_test_ext.shape, y_test_ext.shape)
np.save('X_test_ext',X_test_ext)
np.save('y_test_ext',y_test_ext)
In [253]:
X_test_ext.shape = (len(X_test_ext), 32, 32, 1)
X_valid_ext.shape = (len(X_valid_ext), 32, 32, 1)
X_train_ext.shape = (len(X_train_ext), 32, 32, 1)
np.save('X_test_ext',X_test_ext)
np.save('y_test_ext',y_test_ext)
np.save('X_train_ext',X_train_ext)
np.save('y_train_ext',y_train_ext)
np.save('X_valid_ext',X_valid_ext)
np.save('y_valid_ext',y_valid_ext)